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Abstract 

We  analyze  a  numerical  scheme  for  solving  the  consumption  and  investment  allocation 
problem  studied  in  a  companion  paper  by  Hindy,  Huang,  and  Zhu  (1993).  For  this  prob- 
lem, Bellman's  equation  is  a  differential  inequality  involving  second  order  partial  differ- 
ential equation  with  gradient  constraints.  The  solution  involves  finding  a  free-boundary 
at  which  consumption  occurs.  We  prove  that  the  value  function  is  the  unique  viscos- 
ity solution  to  Bellman's  equation.  We  describe  a  numerical  analysis  technique  based 
on  Markov  chain  approximation  schemes.  We  approximate  the  original  continuous  time 
problem  by  a  sequence  of  discrete  parameter  Markov  chains  control  problems.  We  prove 
that  the  value  functions  and  the  optimal  investment  policies  in  the  sequence  of  the  ap- 
proximating control  problems  converge  to  the  value  function  and  the  optimal  investment 
policy,  if  it  exists,  of  the  original  problem.  We  also  show  that  the  optimal  consumption 
and  abstinence  regions  in  the  sequence  of  approximating  problems  converge  to  those  in 
the  original  problem. 


"Hindy  is  at  the  Graduate  School  of  Business,  Stanford  University,  Stanford,  CA  94305.  Huang 
is  at  the  Sloan  School  of  Management,  Massachusetts  Institute  of  Technology,  Cambridge,  MA  02139. 
Zhu  is  with  the  Derivatives  Research  Division  at  CitiBank.  We  acknowledge  financial  support  from  the 
National  Science  Foundation  under  grants  NSF  SES-9022937  and  NSF  SES-9022939. 

1 


1     INTRODUCTION  1 

1      Introduction 

We  analyze  a  numerical  scheme  for  solving  the  problem  of  optimal  life-time  consumption 
and  investment  allocation  studied  in  a  companion  paper  by  Hindy,  Huang,  and  Zhu 
(1993),  denoted  HH&Z  henceforth.  HH&Z  study  a  model  that  represents,  among  other 
things,  habit  forming  preferences  over  the  service  flows  from  irreversible  purchases  of  a 
durable  good  whose  stock  decays  over  time.  We  provide  a  brief  statement  of  a  special 
case  of  the  control  problem  in  section  2.  HH&Z  contains  the  complete  formulation,  the 
motivation  of  the  study,  the  economic  interpretations  of  the  variables,  and  a  detailed 
discussion  of  the  numerical  solution.  In  this  paper,  we  only  provide  the  theoretical 
framework  that  supports  the  ainalysis  in  HH&Z. 

We  use  dynamic  programming  to  solve  the  optimal  stochastic  control  for  the  contin- 
uous time  problem  formulated  in  HH&Z.  The  important  feature  of  this  problem  is  that 
the  admissible  controls  allow  for  cumulative  consumption  processes  with  possible  jumps 
and  singular  sample  paths.  Furthermore,  the  controls  do  not  appear  directly  in  the  ob- 
jective functional.  As  a  result,  the  associated  Bellman's  equation,  formula  (7)  in  the 
sequel,  takes  the  form  of  a  differential  inequality  involving  a  nonlinear  second  order  par- 
tial differential  equation  with  gradient  constraint.  The  differential  inequality  reveals  the 
"free-boundary"  nature  of  the  solution.  In  particular,  consumption  occurs  only  at  some 
boundary  surface  in  the  domain  of  definition  of  the  value  function.  This  free-boundary 
is  the  area  in  the  state  space  at  which  both  the  second  order  differential  equation  and 
the  gradient  constraint  are  equal  to  zero.  Furthermore,  the  optimal  consumption  is  the 
amount  required  to  keep  the  trajectory  of  the  controlled  process  from  crossing  the  free- 
boundaxy.  This  typically  results  in  optimal  consumption,  and  associated  wealth,  with 
singular  sample  paths;  hence  the  term  "singular  control". 

In  section  3.1,  we  state  the  differential  inequality  that  the  value  function  solves. 
HH&Z  contains  the  details  of  the  derivation  which  is  heuristic  because  the  value  function 
is  assumed  to  be  twice  continuously  differentiable.  Under  the  same  assumption,  Hindy 
and  Huang  (1993)  and  HH&Z  show  that  if  a  function  solves  the  differential  inequality, 
together  with  boundary  and  regularity  conditions,  then  such  a  function  is  indeed  the 
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value  function.  Furthermore,  HH&Z  provide  a  verification  theorem  to  check  whether  a 
candidate  policy  is  optimal. 

In  general,  it  is  difficult  to  establish  smoothness  of  the  value  function.  To  do  so, 
one  can  either  rely  on  analytical  results  characterizing  the  regularity  of  the  solution  of 
Bellman's  equation  or  produce  a  closed  form  expression  for  the  value  function  whose 
regularity  can  be  verified.  Neither  option  is  available  in  our  case.  In  paxticulax,  standard 
results  in  the  theory  of  nonlinear  partial  difFerenticd  equations;  see,  for  example,  Friedman 
(1964)  and  Ladyzenskaja,  Solonnikov,  and  Ural'ceva  (1968),  do  not  provide  evidence  for 
the  smoothness  of  the  solution  to  our  differential  inequality  with  its  gradient  constraints. 
We  will  show,  however,  in  section  3  that  the  diflferentiaJ  inequality  has  a  solution  in  the 
so-caJled  viscosity  sense.  Furthermore,  that  solution  is  indeed  the  value  function  of  our 
dynamic  program.  We  will  define  the  notion  of  viscosity  solutions  in  section  3. 

In  section  4,  we  provide  the  details  of  the  numerical  scheme  used  for  obtaining  the 
solutions  reported  in  HH&Z.  The  numerical  scheme  we  use  is  based  on  the  Markov  chain 
approximation  method  introduced  by  Kushner  (1977)  and  described  in  details  in  Kushner 
and  Dupuis  (1992).  We  approximate  the  original  controlled  continuous  time  processes 
by  a  sequence  of  controlled  Markov  chains.  We  peirameterize  the  sequence  of  controlled 
Markov  chains  by  the  size  of  the  possible  jump  in  each  chain  which  we  denote  by  h.  The 
approximating  Markov  chains  are  chosen  such  that,  as  /i  |  0,  the  mean  and  mean  square 
change  per  step,  under  any  control,  resemble  more  and  more  closely  the  local  drift  and 
the  local  variance,  respectively,  of  the  controlled  continuous  time  process.  This  property 
is  called  "local  consistency".  We  also  replace  the  objective  functional  for  the  original 
problem  by  a  sequence  of  appropriate  anailogs  for  the  Markov  chains. 

Given  the  chosen  approximating  Meirkov  chains  and  objective  functional,  the  ap- 
proach is  to  approximate  the  original  continuous  time  problem  by  a  sequence  of  discrete 
parameter  Markov  decision  problems.  For  each  decision  problem  in  the  sequence,  the 
corresponding  value  function  satisfies  an  iterative  Bellman's  equation.  Such  iterative 
equations  are  easy  to  program  <ind  solve  numerically.  Furthermore,  the  sequence  of 
discrete  Bellman's  equations  can  be  viewed  as  finite-difference  approximations  of  the 
original  Bellman  equation  for  the  continuous  time  case. 
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—  Next,  we  turn  our  attention  to  the  convergence  of  the  numerical  scheme.  In  particu- 
lar, we  focus  on  convergence  of  the  value  functions  in  the  sequence  of  the  approximating 
decision  problems  to  the  value  function  of  the  original  continuous  time  problem.  We  use 
analyticaJ  techniques,  in  particular  the  notion  of  viscosity  solutions,  to  address  this  ques- 
tion. The  argument  runs,  roughly,  as  follows.  We  demonstrate  that  the  vaJue  function  is 
the  unique  viscosity  solution  of  Bellman's  equation.  This  is  achieved  in  section  3  in  two 
steps.  We  first  show  that  the  value  function  is  a  viscosity  solution.  We  next  develop  a 
comparison  result  between  sub-  and  super-solutions,  defined  in  section  3,  and  use  it  to 
establish  the  uniqueness  result.  \ 

We  then  focus  on  the  sequence  of  Bellman's  equations  for  the  approximating  Markov 
chain  problems  and  consider  those  as  finite-difference  approximations  to  the  original 
Bellman's  equation.  We  establish  that  this  sequence  of  finite-difference  approximations  is 
monotone,  consistent,  aad  stable.  We  will  define  these  notions  in  section  5.1.  Using  these 
properties,  we  show  that  the  sequence  of  value  functions  for  the  approximating  Markov 
decision  problems  converges  to  a  viscosity  solution  of  the  original  Bellman's  equation.  The 
uniqueness  result  established  in  section  3  implies  that  this  limit  is  the  value  function  for 
the  original  program.  We  present  the  convergence  analysis  in  section  5.  In  that  section, 
we  also  prove  that  the  sequence  of  optimal  investment  policies  in  the  approximating 
decision  problems  converges  to  the  optimal  investment  policy  in  the  original  problem  if  the 
latter  indeed  exists.  Furthermore,  we  show  that  the  sequence  of  regions  of  consumption 
and  abstinence  in  the  approximating  problems  converge  to  the  corresponding  ones  in  the 
original  problem. 

It  is  perhaps  worthwhile  to  point  out  that  the  analysis  in  this  paper  pays  special  atten- 
tion to  the  possibilities  of  jumps  and  singular  sample  paths  in  the  admissible  consumption 
policies,  and  the  special  structure  of  Bellman's  equation  imposed  by  the  presence  of  the 
gradient  constraints.  In  particular,  the  proofs  of  theorems  1  to  4  require  involved  argu- 
ments to  handle  the  possible  jumps  in  consumption  and  the  gradient  constraints.  These 
arguments  extend  the  existing  literature  on  viscosity  solutions  in  dynamic  programming 
as  reported,  for  example,  in  Fleming  and  Soner  (1993). 
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2      Statement  of  the  Control  Problem 

In  this  section,  we  state  the  control  problem  which  is  the  focus  of  analysis  in  this  paper. 
This  problem,  with  constant  investment  opportunity  set  and  infinite  horizon,  is  a  special 
case  of  the  general  problem  studied  by  HH&Z.  For  a  statement  of  the  general  formulation, 
a  complete  discussion  of  the  motivations  for  studying  this  problem,  and  for  the  economic 
interpretation  of  the  variables,  we  refer  the  reader  to  Hindy  and  Huang  (1993)  and  HH&Z. 

2.1      Financial  Market 

We  consider  a  frictionless  securities  market  with  two  long  lived  and  continuously  traded 
securities:  a  stock  and  a  bond.  The  stock  is  risky,  pays  no  dividends,  and  sells  for  S{t) 
at  time  t.  The  bond  is  riskless,  does  not  pay  dividend,  and  sells  for  B{t)  =  e'"'  at  time  t, 
where  r  is  the  constant  riskless  interest  rate. 

The  price  process^  for  the  stock  is  a  geometric  Brownian  Motion  given  by:^ 

S{t)  =  S{0)+  [  fiS{s)ds+  f' aS{s)dB{s)     Vi>0     a.s.,  (1) 

Jo  Jo 

where  B  is  one-dimensional  standard  Brownian  Motion  defined  on  a  complete  probability 
space  {Q,^,P)  and  fi  >  r  and  cr  >  0  are  constants.  Note  that  the  prices  of  both  the 
stock  and  bond  cire  strictly  positive  with  probability  one.  The  available  information  in 
the  market  is  given  by  F  =  {^t;t  €  [0,oo)}  where  ^t  is  the  smallest  sub-sigma-field  of 
J^  with  respect  to  which  {5(5);  0  <  5  <  i}  is  measurable.  All  processes  to  be  discussed 
will  be  adapted  to  F.^ 


'A  process  y  is  a  mapping  Y:Q  x  [0,oo)  — ►  3J  that  is  measurable  with  respect  to  ^ <Si  B{[0,oo)),  the 
product  sigma-field  generated  by  ^  and  the  Borel  sigma-field  of  [0,  oo).  For  each  u  £Q,  Y{u, .):  [0,  oo)  — ► 
3t  is  a  sample  path  and  for  each  t  €  [0,  oo),  Y{.,  <):  Q  — ►  3?  is  a  random  variable. 

^The  notation  a.s.  denotes  statements  which  are  true  with  probability  one.  We  will  use  weak  relations. 
Increasing  means  nondecreasing  and  positive  means  nonnegative. 

^We  assume  that  Tt  is  complete  in  the  sense  that  it  contains  all  the  probability  zero  sets  of  J-.  The 
process  Y  is  said  to  be  adapted  to  F  if  for  each  t  6  [0,oo),  Y{t)  is  ^,-measurable. 


2    STATEMENT  OF  THE  CONTROL  PROBLEM  5 

2.2  Admissible  Controls 

The  control  variables  are  a  consumption  plan  and  an  investment  strategy. -A- consumption 
plan  C  is  a  process  whose  seimple  paths  are  positive,  increasing,  and  right- continuous, 
with  C{u},  t)  denoting  the  cumulative  consumption  from  time  0  to  time  /  in  state  u;.  An 
investment  strategy  is  one-dimensional  process  A,  where  A{t)  denotes  the  proportion  of 
wealth  invested  in  the  stock  at  time  t  before  consumption  and  trading.  A  consumption 
plan  C  is  said  to  be  financed  by  the  investment  strategy  A  if,  a.s., 

W{t)  =  W{0)+  I  {rW{s)  +  A{s){ti-r)W{s))  ds-C{t-)  +  f  aA{s)W{s)dB{3),  V<  >  0, 
^0  Jo 

(2) 
where  W{t)  is  wecdth  at  time  t  before  consumption.  Note  that  the  wealth  process  has 
left-continuous  sample  paths  and  that  H^(f'")  =  W{i)  —  AC(<). 

A  consumption  plan  C  and  the  investment  strategy  A  that  finances  it  axe  said  to  be 
admissible  if  (2)  is  well-defined^  and,  for  all  t,  E[C{t)]  <  Q{t),  with  Q{t)  some  polynomial 
of  t.  Denote  by  C  and  A  the  spjice  of  admissible  consumption  plans  and  trading  strategies, 
respectively. 

2.3  Objective  Functional 

The  objective  functional  is  defined  as  an  integraJ,  over  time,  of  a  utility  function  defined 
over  the  stock  of  durable,  z,  and  the  standard  of  living,  y,  which  are  given,  respectively, 

by 

z{t)    =    z{0-)e-'^' + 13  f\-'^^'-'UC{s)     a.s.,  (3) 

Jq- 

y{t)    =    y(0-)e-^'  +  A  /'  e-^('-')ffC(5)     a.s.,  (4) 

no- 
where 2(0")  >  0  and  y(0")  >  0  are  constants,  /?  cind  A,  with  0  >  X,  are  weighting  factors, 
and  the  integrjJs  in  (3)  and  (4)  are  defined  path  by  path  in  the  Lebesgue-Stieltjes  sense. 


*For  this  we  mean  both  the  Lebesgue  integral  and  the  ltd  integral  are  well-defined.  When  A(t)  is  a 
feedback  control  depending  on  the  state  variables  and  C{t)  depends  on  the  history  of  {W,S),  we  mean 
there  exists  a  solution  W  to  the  stochastic  differential  equation  (2)  for  every  pair  of  feedback  controls. 
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Wealth  is  managed  dynaimicaJly  to  solve  the  following  program:  ••  - 

supc€C    E[l^e-''u{y{t),z{t))dt] 

s.t.      C  is  financed  by  A  e  A  with  W{0)  =  Wq,  (5) 

and  W{t)  -  AC{t)  >  0     V<  >  0, 

where  Wq  is  initial  wealth,  <5  >  0  is  a  discount  fcictor  and  u(-,-)  is  the  felicity  function. 
The  function  u  is  cissumed  to  satisfy  the  following  conditions: 

•  u  is  nonnegative  and  satisfies  the  linear  growth  condition: 

0  <  «(y,  z)  <  K{y  +  z)-,         Vj/  >  0,  z  >  0.     for  some  constant  /T  >  0.         (6) 

•  For  a  fixed  y,  u{y,  •)  is  a  strictly  increasing,  concave,  and  differentiable  function. 

•  For  a  fixed  z,  u(-,z)  is  a  strictly  increasing,  concave,  and  differentiable  function. 

The  second  constraint  in  (5)  is  the  requirement  that  wealth  after  consumption  at  any 
time  must  be  positive.  Condition  (6)  guarantees  that  the  supremum  in  (5)  is  finite. 

Since  the  interest  rate  and  the  parameters  of  the  return  on  the  stock  are  constant, 
the  supremum  in  (5)  is  a  function  of  the  state  variables  W ,  y,  and  z  only.  As  usual,  we 
call  the  supremum  in  (5)  the  value  function  of  the  dynamic  program  and  denote  it  by 
J(H^,  y,z).  We  observe  that,  in  general,  little  is  known  about  the  differentiability  of  J. 
The  choice  of  a  technique  to  analyze  the  dynamic  program  in  (5)  is  influenced  by  prior 
knowledge  of  the  smoothness  of  the  value  function. 

3      Analysis  of  The  Value  Function 

3.1       Bellman's  Equation 

In  this  section,  we  report  Bellman's  equation  associated  with  the  dynamic  program  in 
(5).  Although  little  is  known  about  the  regularity  of  the  value  function  J,  we  can  derive 
Bellman's  equation  heuristicaJly  by  assuming  that  J  is  twice  continuously  differentiable. 
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This  assumption  allows  us  to  utilize  Ito's  lemma,  together  with  the  dynamic  programming 
principle,  to  derive  Bellmaui's  equation.  Bellman's  equation  turns  out  to  be  a  difFerentiaJ 
inequality  involving  a  nonlinear  second  order  partial  differential  equation  with  gradient 
constraints.  In  section  3,  we  prove  rigorously  that  the  value  function  is  indeed  a  solution 
of  Bellman's  equation  we  derive  heuristically  here.  The  value  function,  however,  is  a 
solution  in  the  viscosity  sense —  a  notion  that  we  define  in  section  3. 

Assuming  that  the  value  function  is  smooth,  HH&Z  show  that  the  necessary  condi- 
tions for  optimality  are  given  by  the  following  differential  inequality: 


(7) 


max|max^[u(y,2)  +  JwW[r  +  A{fi  -  r)]  +  ^JwwW'^A'^a  -  /3zJ^  -  XyJy  -  SJ] , 

M  +  AJv  -  Jw}  =  0, 

plus  the  condition  that  nontrivial  consumption  occurs  only  at  states  where  /3 J,  -}-  XJy  = 
Jw-  The  differential  inequjJity  in  (7)  can  also  be  viewed,  in  the  region  of  no  consumption, 
as  a  nonlinear  second-order  partial  differential  equation  with  the  gradient  constraint  that 
13 J,  +  XJy  -Jw<0. 

In  addition  to  the  above  necessary  conditions,  it  is  clear  that  J  must  satisfy  the 
following  boundary  condition: 

^(y,z)  =  \unJ{W,y,z)  =  j^  e-"u{ye-'\ze-^')ds  .  (8) 

The  differential  inequzJity  (7)  suggests  that  the  dynamic  program  formulated  in  sec- 
tion 2  is  a  free-boundciry  singulcir  control  problem.  Consumption  should  occur  only  when 
the  state  variables  {W^y^z)  reach  a  certain  boundary  surface  characterized  by  the  condi- 
tion that  Jw  —  l3Jz  +  XJy.  Furthermore,  the  optimal  consumption  is  the  amount  required 
to  keep  the  trajectory  of  the  controlled  processes  from  crossing  this  critical  boundary. 
Since  the  returns  on  the  stock  follow  a  Brownian  Motion,  such  a  consumption  policy, 
and  the  associated  wealth  process,  may  have  singular  sample  paths —  hence  the  term 
"singular  control".  Computing  the  location  of  the  free-boundary  is  the  focus  of  this 
paper. 

Hindy  and  Huang  (1993)  and  HH&Z  show  that  if  there  is  a  smooth  solution  to  (7) 
that  satisfies  (8)  and  additional  mild  regularity  conditions,  then  that  solution  is  the 
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value  function  for  the  program  (5).  These  papers  also  provide  verification  theorems 
that  specify  conditions  for  a  candidate  feasible  consumption  and  investment  policy  to 
attain  the  supremum  in  (5).  We  refer  the  reader  to  HH&Z  (§3)  for  the  statements  of 
the  necessary  and  sufficient  conditions  of  optimality,  the  verification  theorems,  and  the 
proofs. 

3.2      Viscosity  Solutions  of  The  Dynamic  Programming  Equa- 
tion 

The  verification  theorem  presented  in  HH&Z  is  useful  when  it  is  known  that  the  value 
function  is  twice  continuously  differentiable.  Such  is  the  ca^e  when  there  is  a  closed  form 
solution  of  Bellman's  equation  as  in  Hindy  and  Huang  (1993).  In  general,  it  is  difficult 
to  know  whether  the  value  function  has  enough  regularity  for  the  verification  theorem  to 
apply.  To  handle  the  situations  in  which  there  is  no  information  about  the  regularity  of 
the  value  function,  we  utilize  the  notion  of  weak,  or  viscosity,  solutions. 

Viscosity  solutions  were  introduced  by  Crandall  and  Lions  (1982)  for  first-order  equa- 
tions, and  by  Lions  (1983)  for  second-order  equations.  For  a  general  overview  of  the 
theory,  we  refer  the  reader  to  the  User's  Guide  by  Crandall,  Ishii,  and  Lions  (1991). 

We  are  interested  in  viscosity  solutions  in  the  context  of  proving  convergence  of  the 
Markov  chain  numerical  scheme  to  be  presented  in  the  following  sections.  To  that  end, 
we  characterize  the  value  function  as  the  unique  viscosity  solution  of  Bellman's  equation. 
We  prove  uniqueness  of  the  solution  to  Bellman's  equation  by  utilizing  a  comparison 
principle  between  viscosity  subsolutions  and  viscosity  supersolutions.  This  comparison 
result  has  additional  significance  to  our  subsequent  analysis  because  it  is  a  key  element 
in  establishing  convergence  of  the  numerical  scheme. 

Let  Q  =  (0,  oo)  X  [0,  oo)  x  [0,  oo)  and  consider  a  second-order  partial  differential 
equation  of  the  form: 

ma.x[-6v  +  F{d^v,  Dv)  +  u{y,  z),  7  •  D^ v)  =  0,    on  Q     with  u(0,  y,  z)  =  $(y,  2) ,  (9) 

where  7  =  (  — l,A,/3),  D  =  {dw,dy,dz),  $  is  some  function  of  y  and  z,  and  ^  denotes 


3    ANALYSIS  OF  THE  VALUE  FUNCTION 


transpose.  The  function  F  is  given  by: 


F{dwv,  dwv,  dyV,  d^v)  =  max{- VK^A^a^5^u  +  W[r  +  A{fi  -  r)]dwv]  -  \ydyV  -^zd^v. 

Next,  we  state  the  notion  of  viscosity  solutions  as  in  Crandall,  Ishii,  and  Lions  (1991). 

Definition  1  (Viscosity  Solution)  (i)  An  upper  semicontinuous  function  v  :  Q  —^  R 
is  a  (viscosity)  subsolution  of  (9)  on  the  domain  Q  if  v{0,y,z)  <  $(j/,z)  and  for  every 
4>  G  C^'^'^{Q)  and  any  local  maximum  point  (VKo,yo,  2o)  ^  Q  of  v  ~  (j), 

max|-(5v(iyo,  I/o,  ^o)  +  F{d1y<i>{Wo,  yo,  zo),  4>w{Woyyo-,  2o),  <?!'y(VKo,  j/o,  ^o),  <i>z{Wo,  yo,  ^o)) 

+u(yo,  zo),  7  •  D^<i>{Wo,  yo,  zo)}  >  0  . 

(10) 

(ii)  A  lower  semicontinuous  function  v  :  Q  —*  R  is  a  (viscosity)  supersolution  of  (9)  on 
the  domain  Q  if  v{0,y,  z)  >  ^{y,z)  and  for  every  (f>  6  C^'^'^{Q)  and  any  local  minim.um 
point  (Wo,yo,zo)  e  Q  ofv-4>, 

max|-(^u(V^o,  yo,  zo)  +  F{d^(^{Wo,  yo,  zo),  <f>w{WQ,  yo,  zq),  <f>y{Wo,  yo,  zo),  <f>z{Wo,  yo,  Zo)) 

+u(yo,Zo),7--0''<?i(Wo,yo,Zo)}  <0. 

(11) 
A  continuous  function  v  :  Q  —^  R  is  a  viscosity  solution  of  (9)  if  it  is  both  a  subsolution 

and  a  supersolution  of  (9). 

The  main  result  of  this  section  is  that  J(W,y,z)  is  the  unique  viscosity  solution  of 
Bellman's  equation  (9)  with  boundary  condition  u(0,y,z)  =  ^(y,2)  as  in  (8).  First,  we 
state  without  proof  the  following  proposition 

Proposition  1  (Optimality  Principle)  The  value  function  J  satisfies  the  dynamic 
programming  principle 

JiW,y,z)=      sup      E[e-'^J{WiT),y{T),z{T))+  re-''u{y{t),z{t))dt],  (12) 

for  any  Ft-adapted  stopping  time  r  >  0,  where  W{t),  y{T),  and  z{t)  are,  respectively, 
the  stopped  processes  ofW,y,  and  z  at  r. 
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The  dynamic  programming  principle  in  proposition  1  is  similar  to  what  can  be  found, 
for  example,  in  Fleming  and  Soner  (1993,  §5.2).  The  proof  of  proposition  1  is,  however, 
slightly  different  from  the  standard  proofs  because  in  our  formulation  the  admissible 
controls  allow  for  jumps  and  singular  sample  paths.  A  proof  of  the  dynamic  programming 
principle  in  such  a  case  can  be  found  in  Zhu  (1991,  Chapter  II).  Next,  we  show  how  one 
may  utilize  the  principle  of  dynamic  programming  to  prove  that  the  value  function  is  a 
viscosity  solution  of  Bellman's  equation.  The  proof  of  this  result  is  not  standard  because 
of  the  possible  jumps  and  the  singular  sample  paths  in  the  admissible  controls. 

Theorem  1    The  value  function  J  is  a  viscosity  solution  of  Bellman's  equation  (9). 

Proof.  We  first  establish  that  the  vcJue  function  J  is  continuous  on  the  domain 
Q.  Utilizing  the  concavity  of  u,  and  using  standard  reasoning,  we  can  show  that  J 
is  concave  in  all  of  its  arguments.  It  then  follows  that  J  is  continuous  in  the  region 
(0,  oo)  X  (0,  oo)  X  (0,  oo);  see  Holmes  (1975,  §14,  page  82).  We  then  need  to  show  that  J  is 
continuous  on  the  boundary  (IV,  0, 0),  W  G  (0,  oo),  of  the  doamin  Q.  Since  u  is  increasing, 
the  value  function  J  is  increasing  in  all  of  its  elements,  and  hence  limy_o,z-.o  J{^i  V-,  2)  > 
J(H^,0,0).  We  prove  that,  in  fact,  Yimy^Q,,^oJ{W,y,z)  =  J(W^,0,0). 

Suppose  that  limy—o,z-.o  «^(^)  J/,^)  >  J(Vl^,  0,0).  From  the  dynamic  programming 
principle,  we  have 

J{W,  0, 0)  >  i;  [e-''J{Wu  yu  ~zt)  +  f^  e-''u{y„  z,)  ds]  ,  (13) 

where  (M^«,yj,2j)  are  the  trajectories  of  a  continuous  state  process  for  some  admissible 
controls.  Taking  the  limit  as  <  J,  0,  and  utilizing  the  continuity  of  the  trajectory  of  the 
controlled  process,  the  right-hand  side  of  (13)  converges  to  a  value  strictly  greater  than 
J{W,0,0)-  a  clear  contradiction. 

We  next  show  that  J  is  a  viscosity  solution  of  Bellman's  equation. 
(i)  J  is  a  viscosity  subsolution.  We  argue  by  contradiction  and  show  that  if  J  fails  to  be  a 
subsolution,  then  the  dynamic  programming  principle  stated  in  proposition  1  is  violated. 
Let  (^  be  a  C^'*'^((5)  function  such  that  J  —  <l>  has  a  local  maximum  at  {Wq,  yo,  zq)  6  Q. 
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That  is,  3  a  neighborhood  A/q  of  {Wo,yo,zo)  such  that 

J{Wo,yo,zo)  -  HWo.yo^zo)  >  J{W,y,z)  -  4>{W,y,z),         y{W,y,z)  e  Afo- ■       (14) 

Without  loss  of  generality,  we  assume  that  J(VFo,yo,2o)  =  <^(^o?  J'o,  ■^o)-  Now  suppose 
that  <j>  violates  inequality  (10).  In  other  words,  both  \<f>y  +  ^(f>z  —  <f>w  <  0  and  —SJ  + 
F{<f>ww,4>w,4>y,4>z)  +  '^{yi^)  <  0  at  {Wo,yoiZo).  By  continuity,  and  using  the  assumption 
that  J{Wo,yo,ZQ)  =  <f){Wo,yo,Zo),  there  exists  7  >  0  such  that,  for  {W,y,2)  in  some 
neighborhood  A/  of  (Wo, J/0,20), 

X<f>y{W,y,z)  +  l3<f>,{W,y,z)-4>w{W,y,z)<0     and  (15) 

-  6<f>{W,  y,  z)  +  F{dU{^,  y.  ^),  4>w{W,  y,  z),  <f>,{W,  y,  z),  M^,  y,  z))  +  u{y,  z)  <  -7 . 

(16) 

By  the  existence  result  in  Zhu  (1991,  Theorems  IV. 2. 2  and  IV. 2. 3),  we  consider  the 

optimal  policy  {A',C')  at  (Wo,yo,  ■^o)-^    We  now  establish  that  the  optimal  policy  C 

does  not  prescribe  an  initial  jump  at  {Wo,yo,Zo).  If  not,  then  C  prescribes  some  jump 

T]o.  Using  the  dynamic  programming  principle,  we  can  conclude  that 

^(Vl^o,yo,2o)  < -^(Vl^o-'?,yo  + A7?,Zo  +  /?r/),  VO  <  r/ <  770 . 

By  the  property  of  the  local  maximum  in  (14),  the  previous  inequality  also  holds  for  ^, 
and  hence 

0  <  (i>{Wo  -  r/,  yo  +  At?,  zq  +  /9r?)  -  4>{^o,  Vo,  Zo) 

where  0  <  t;  <  7?o  is  sufficiently  small  so  that  (Wo  —  t?,  yo  +  A77,  zo  +  0t])  €  A/q.  Dividing 
by  T]  and  letting  77  J.  0,  we  obtain 

0  <  X4>y{Wo,  yo,  Zo)  +  ^4>ziWo,  yo,  zo)  -  <i>w{Wo,  yo,  zq)  , 


^From  the  analysis  in  Zhu  (1991,  Chapter  IV),  the  optimal  policy  {A'  ,C')  is  known  to  exist.  However, 
{A' ,  C)  may  be  realized  in  a  space  of  processes  larger  than  the  space  of  admissible  controls  {A,  C).  The 
crucial  feature  of  the  analysis  is  that  the  expected  life  time  utility  associated  with  {A',C'),  starting 
from  any  point  [W,  y,  z),  is  equal  to  the  viJue  function  J  evaluated  at  that  point.  For  more  details,  we 
refer  the  reader  to  Zhu  (1991). 
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which  is  a  contradiction  to  (15). 

We  have  now  established  that  if  (15)  holds,  then  the  optimally  controlled  process 
{W*,y*,Zf)  is  continuous  in  a  neighborhood  of  (WoiJ/Oi^o).  We  next  show  that  if  J  fails 
to  be  a  subsolution,  then  the  dynamic  programming  principle  is  violated. 

We  define  the  optional  time  f  =  inf{i  >  0  :  (VF,',^^*,  2j')  ^  Ao}.  Clearly,  f  >  0,  a.s., 
by  the  established  fact  that  {W^,y',z^)  is  a.s.  continuous  up  to  r  under  the  optimal 
policy.  Then,  it  follows  from  (15)  and  (16)  that 

E  ll^  e-*'[-<5<^  +  F{dU,  4>w,  K  4>z)  +  u](W:,  yl  z;)dt]  (17) 

+E[j\-'\\<i>,  +  ^<t>,-<i>w]{W:,ylz;)dC^]     <     -7E[f]. 

Define  the  optional  time  f  =  ini{t  >  0  :  Jq  \AlW^<7\^ds  >  K},  for  some  constant 
K.    Now  take  r  to  be  the  minimum  of  f  and  f.     Applying  Ito's  rule  to  the  process 

e-*V(M^;,yr,2;),wehave 

<i>{^o,yo,zo)  =  E[e-''<f>iw:,y;,z;)] 

-E\I^  e-''[-S4>  +  FidU.  4>w^  <f>y,  <f>.)]iW;,  y;,  z:)dt]         (18) 

-E  [J^^  e-^m, + 134>,  -  cf>w]{w;,y:,  z:)dc:] . 

Note  that  the  expectation  of  the  stocheistic  integral  Jq  A'W'a  dBt  vanishes  because  that 
integral  is  a  martingale  as  the  integrand  is  boimded  till  the  stopping  time  r.  Again  by 
the  local  maximum  property  in  (14),  we  can  replace  (^  by  J  in  first  line  of  (18)  and  then 
combine  the  result  with  (17)  to  conclude  that 

J{Wo,yo,zo)    >    E[e-'W{W:,y;,z;)] 

-E[j\-''[\<i>y  +  P4>,  -  4>w]{w:,y;,z;)dc:] 

>    E[J\-''n{ylz:)dt  +  e-'^JiW:,y;,z:)]  +  T^H  , 
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which  clearly  violates  the  optimality  of  (A',C')  ajid  the  principle  of  dynamic  prograxn- 
ming  (12). 

(a)  J  is  a  viscosity  supersolution.  By  definition,  we  must  show,  for  every  C^'^'^- 
smooth  4)  such  that  J(Wo,yo,2o)  -  <?^(W^o,  J/o,2o)  <  J{W,y,z)  -  4>{W,y,z)  for  {W,y,z)  G 
A/q,  that  the  following  holds 

[-8J +  F{4>wwAwAyAz)^-u\{WQ,yQ,zo)     <     0  and  (19) 

A<?^„(H^o,yo,2o)  +  /3<^z(V^o,yo,2o)-<^w(V^o,yo,2o)     <    0.  (20) 

First,  we  choose  zero  consumption  C,  =  0  and  some  investment  policy  A  on  the  stochastic 
interval  [0,t),  where  r  =  inf{<  >  0  :  /q  |i4,W,a|^<i3  >  K}  for  some  constant  ii'  >  0.  Let 
W°,  y°,  and  2°  be  the  state  processes  corresponding  to  the  controls  C,  =  0  and  A  on 
[0,r).  By  the  principle  of  dynamic  progrctmming,  it  follows  that,  for  all  p  <  t,  where  p 
is  an  optional  time, 

J{Wo,yo,zo)>E[e-"'J{W^,y''^,z°)+re-"u{ylz',)ds   . 

Replacing  J  by  <^,  we  then  apply  Ito's  rule,  together  with  the  observation  that  the 
stochastic  integral  is  a  maxtingaJe,  to  obtain  from  the  above  inequality  that 

eIJ'  e-^'[-6<^  +  F  +  u]{W^,ylz'',)ds\  <0     VO  <  p  <  r  . 

Dividing  by  p  and  letting  p  I  0,  using  the  continuity  of  the  integrand,  we  derive  the  first 
inequality  (19).  To  derive  (20),  the  dynamic  programming  principle  implies  that 

J{Wo,  yo,  ^o)  >  J{Wo  -  r/,  yo  +  Ary,  zq  +  ^77),  VO  <  r/  <  Wq. 

Since  J  —  <i>  has  a  local  minimum  at  (Wo,yo,  20)1  this  inequality  also  holds  for  <i>  as  well 
if  7;  >  0  is  sufficiently  small  such  that  {Wq  —  T],yo  +  \t],zq  +  0t])  €  Ao-  Therefore, 
subtracting  the  right-hand  side  from  the  left-hand  side  and  dividing  by  t]  and  sending 
77  J,  0,  we  obtain  the  second  inequality  (20).  This  completes  the  proof.       I 
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3.3      Comparison  of  Subsolutions  and  Supersolutions 

We  conclude  this  section  by  providing  a  theorem  that  compares  the  subsolutions  of  Bell- 
maji's  equation  to  its  supersolutions.  The  proof  of  this  result  is  not  a  simple  extension  of 
standard  results  as,  for  example,  in  Fleming  and  Soner  (1993,  §5.8).  We  utilize  additional 
arguments  to  handle  the  gradient  constraint  that  Jw  —  ^Jy  —  ^Jz  >  0  in  Bellman's  equa- 
tion. The  comparison  result  is  then  used  to  argue  that  the  value  function  is  the  unique 
viscosity  solution  of  the  dynamic  programming  equation.  In  addition,  the  comparison 
result  is  later  used  in  the  proof  of  convergence  of  the  Markov  chain  approximations  to  be 
presented  in  a  subsequent  section. 

Theorem  2  (Comparison  Principle)  Let  V  and  V  be,  respectively,  a  suhsolution  and 
a  supersolution  of  (9)  on  the  domain  Q  in  the  sense  of  Definition  1,  then  V  <V  on  the 
domain  Q. 

Proof.  Let  us  denote,  for  simplicity,  x  =  {W,y,z)  and  x  =  {W,y,z).  We  rewrite 
Bellman's  equation  as  ma.x{—6v  +  F{^^^rV,Dxv)  +  u{y,z),j  •  Djv}  =  0,  where  7  = 
(  — 1,  A,/3)  and  Dj  is  the  transpose  of  {dw,dy,dz).  We  construct  a  function  U{W,y,z)  = 
/o°°  e~*'u(j/e~^',  2~^')  dt  —  mo  for  some  constant  mo  >  0.  One  can  easily  verify  that 

-SU  +  F{d^U, DM)  +  «(l/, 2)  >  0    and    \Uy  +  l3U,-Uw>0, 

where  the  second  inequality  follows  from  the  assumption  that  u  is  strictly  increasing  in 
both  y  eind  z. 

Next,  we  will  prove  the  comparison  between  V  and  V^  =  6V  +  {1  —  0)U,  for  all  9,  from 
which  the  proof  of  the  theorem  is  immediate.  For  this  purpose,  we  choose  the  auxiliary 
function 

rl;{x,  x)  =  ||x  -  x\'  +  e{\x\'  +  [x^  V(x,  x)  e  Q  x  Q  , 

where  a  >  0  and  0  <  c  <  1.  Assume  to  the  contrary  that: 

V\x')  >  V(i'),         for  some  x'  G  Q.  (21) 
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By  the  linear  growth  of  V  and  V,  it  is  easy  to  check  that  V*(x)  —  V{x)  —  xp{x,x)  will 
attain  its  maximum  at  a  point  (io,io)  €  Q^Q-  Note  that  (lo,  xq)  depends  on  the  values 
of  a  and  e.  We  will  suppress  this  dependence  in  the  notation.  One  can  prove,  as  in  Zhu 
(1991,  Appendix  A:  proof  of  comparison  principle),  that  a|io  —  io|^  — >  0  as  a  |  oo,  and 
e(|xo|  +  \xo\)  -*  0  as  c  i  0. 

We  will  show  that  supposition  (21)  produces  the  following  contradiction.   From  the 
definition  of  (xo,Xo),  we  have,  for  small  e, 

V\xo)-V{xo)-xl;{xo,Xo)     =        max      V\x)  -  V{x)  -  4>{x,x) 

(z,x)€QxQ 

>    V\x')-V{x')-2e\x'\^>0. 

However,  we  will  show  next  that  the  fact  that  V*  and  V  are  sub-  and  supersolutions, 
respectively,  implies  that  V*(io)  —  V'(xo)  —  V'(io,Xo)  i  0  as  e  J,  0  Jind  a  t  oo,  which  is  a 
clear  contradiction. 

Applying  Definition  1  to  the  subsolution  V^  and  the  super-solution  V,  we  conclude 

that 

rmix{-SV^ +  F{d^rp,Dr7P)  +  u{-),^- Djrp}     >    0,  at  xq  (22) 

max{-<5V'  +  F(-5^V,-^xV') +  «(•), -7- -010}     <    0,,         at  xo  .  (23) 

Now,  since  Dxxk  =  2ex  +  a\x  —  x\  and  D^^  =  2ex  —  a\x  —  x|,  it  follows  from  inequality 
(23)  that 

-  7  •  Djxl^  =  -7  •  [2exo  -  a(xo  -  xq)]  <  0.  (24) 

Moreover,  we  claim  that  in  (22),  7-  [2cxo  +  q(x  —  xq)]  <  0.  If  not,  the  opposite  inequality 
holds.  Subtracting  it  from  (24),  we  conclude  that  2e7  •  (xq  +  xq)  >  0,  Va  >  0 ,  which  is  a 
contradiction  to  the  fcict  that  e(|xo|  +  |xo|)  i  0  as  e  |  0.  Therefore,  we  must  have  in  (22) 
and  (23)  that 

-6V\xo)  +  F(5^V,  ■OrV')  +  u(l/o,  zo)    >    0,         at  xq     and 
-6V{x)  +  F{-dl,ik,-Dsi^)  +  u{yo,zo)     <    0,  at  xq 
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Subtracting  one  from  the  other  yields  the  inequality 

6[V\xo)  -  V{xo)]  <  F(5^V,  D^rP)  -  F(-a^V,  -D,^)  +  u{yo,  20)  -  "(yo,  ^0).       (25) 
Meanwhile,  we  calculate  (Note:  r  =  0  for  simplicity) 

F{d^w^,  D.'iP)    =    max{^V^o'^'<7'  +  ^oAti[a{Wo  -  Wo)  +  2eWo]} 

-Q:['^yo(yo  -  yo)  +  ^zq{zq  -  zq)\ 

F{~dlxP,  -D,rP)    =    max{-^V^o'^'^'  -  WoAfi[-a{Wo  -  Wo)  +  2eWo]} 

A  I  , 

-a[Ayo(yo  -  j/o)  +  ^zo{zo  -  Zo)\ 
where  the  maximizers  can  be  explicitly  computed  as 

\i  ol{Wo  -  Wo)  +  ItWo  ,      T.  _      \Ji  -q(Wo  -  Wo)  +  2tWo 

^°-~<72  ccWo  °"      ^2  aWo 

Substituting  into  (25)  and  rearranging  the  terms,  we  can  verify  that 

S{V\xo)  -  V{xo)\     <    ^cc{Wo  -  Wo)'  -  2ef^iWo  -  Wo)'  -  2e'-^{W^  +  W^) 

-a{X\yo  -  yoT  +  /?ko  -  zoP)  +  [u(yo,  zo)  -  "(yo,  ^o)] , 

which  goes  to  zero  as  c  X  0  and  a  |  00.  This  completes  the  demonstration  of  a  contra- 
diction and  the  proof  of  the  theorem.      I 

Corollary  1  The  value  function  J  is  the  unique  viscosity  solution  of  Bellman's  equation 
(9). 

Proof.  Suppose  that  there  is  another  viscosity  solution  V.  We  then  have  V  <  J  because 
V  and  J  are  sub-  and  supersolutions,  respectively.  On  the  other  hand,  we  have  J  <  V 
because  J  and  V  are  also  sub-  and  supersolutions,  respectively.  Hence,  J  =  V.      I 
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4      Numerical  Approach 

In  this  section  we  describe  the  numerical  approach  utilized  to  solve  the  infinite- horizon 
utility  maximization  problem.  We  replace  the  continuous  time  processes  {W,y,z)  by  a 
sequence  of  approximating  discrete  parameter  Markov  chains.  The  important  feature  of 
the  approximating  sequence  is  that,  as  the  step  size  converges  to  zero,  the  mean  and 
mean  square  change  per  step,  under  any  control,  converge  to  the  local  drift  and  the 
local  variance,  respectively,  of  the  controlled  continuous  time  process.  This  property  is 
called  "local  consistency".  The  original  optimization  problem  is  then  approximated  by 
a  sequence  of  discrete  parameter  control  problems.  We  show  that  the  value  function  of 
each  Markov  chain  control  problem  satisfies  an  iterative,  and  hence  easily  programmable, 
discrete  Bellman's  equation.  The  Bellman's  equation  of  the  discrete  control  problem  is 
a  natural  finite-difference  approximation  to  the  continuous  time  Bellman's  "nonlinear" 
second  order  partial  differential  equation  with  gradient  constraint. 

4.1      A  Constrained  Problem 

For  numerical  implementation,  we  need  to  restrict  the  domain  of  definition  of  the  state 
variables  to  a  finite  region  Qm  =  [0,Mw]  x  [0,My]  x  [0,71/^].  For  this  purpose,  we 
introduce  the  reflection  processes  Lt^Rt  and  Dt  at  the  boundaries  of  the  region.  We 
hence  consider  the  following  modified  state  variables: 

dyt    =     —\ytdt  +  XdCt  —  dLt , 
dzt     =     -^Zidt  +  ^dCt  -  dRt ,      and 
dWt    =    Wt{r  +  Atifi  -  r))dt  -  dCt  +  WtAtadBt  -  dDt , 

where  Lt,  Rt  and  Dt  are  nondecreasing  and  increase  only  when  the  gtate  processes 
{W,z,y)  hit  the  cutoff  boundaries  W  =  Mw,  ^  =  M^,  and  y  =  My,  resp-^  tively. 

We  recall  that  the  stock  price  process  follows  a  geometric  Brownian  I    ^tion  and  that 
the  riskless  interest  rate  is  constant.  We  will,  hence,  apply  the  Markov.^  hain  approach 
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to  solve  numerically,  on  the  restricted  domain,  the  following  infinite  horizon  program: 

/•oo 

j'^{W,y,z)  =  .  sup    E         e-''u{yt,zt)dt  (26) 

A,CGAC       Jo 

subject  to  the  dynamics  of  z  and  y  and  the  dynamic  budget  constraint,  where  {W,  y,  z)  6 
Qm  is  the  initial  state.  The  Bellman's  equation  for  this  dynamic  program  takes  the  form: 

max{-<5J^  +  max{£^  J^}  +  u{y,  2),  Ajf  +  (iJ^  -  J^]  =  0,  (27) 

on  [0,  Mw]  X  [0,  My]  x  [0,  Mz],  together  with  the  appropriate  boundary  conditions,  where 
the  operator  £^  is  given  by  £^  =  ^W^A^a^-^  +  W[r  +  A{fji  -r)]^-  \y§^  -  ^zj-^.  In 
addition,  we  specify  the  boundary  constraints 

__.0,         —  =  0,         and         —  =  0, 

respectively  at  the  cutoff  boundaries  W  =  Mw,  y  =  My,  and  z  =  M^.  This  type  of 
boundary  conditions  was  suggested  in  Kushner  and  Dupuis  (1992,  §5.7)  and  employed 
in  the  numerical  analysis  of  the  one-dimensional  financial  model  studied  by  Fitzpatrick 
and  Fleming  (1992).  These  boundary  constraints  reflect  the  asymptotic  behavior  of 
the  value  function  on  the  original  unbounded  domain.  It  is  worthwhile  to  remark  that 
in  the  theoretical  analysis  of  convergence  of  the  numerical  scheme,  the  value  functions 
J^  converge  pointwise  to  J  as  the  sequence  of  restricted  domains  Qm  increases  to  Q, 
regardless  of  the  specification  of  the  behavior  of  J^  at  the  boundaries.  The  boundary 
conditions  we  chose,  however,  affect  the  quality  of  the  actual  numerical  solution. 

4.2      The  Mcirkov  Chain  Scheme 

We  introdu-e  a  sequence  of  grids 

Gh  =  {{k.       ):W  =  kxh,y  =  ixh,z=j  xh;0<k<  Nw,0<i  <  iV„,0  <j  <  N,}  , 

where  h  is  i.ie  step  size,  and  where  A^y  =  My/h,  N^  =  M^/h,  and  Nw  =  Mw/h  are 
integers.  Each  grid  point  {k,i,j)  £  Gh  corresponds  to  a  state  {W,y,z)  with  W  =  k  x  h, 
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y  =  i  X  h,  and  z  =  j  x  h.  For  simplicity,  we  taJce  the  saime  step  size  in  all  three  directions. 
For  a  fixed  grid,  we  introduce  the  spa<;e  of  discrete  strategies 

ACn  =  {{A,  Cy,A  =  lxA,AC  =  OoTh;0<l<NA} 

where  A  is  the  step  size  of  control,  A'^^  =  A/ A  is  total  number  of  control  steps  with  A 
an  artificial  bound  to  be  relaxed  in  the  limit  eis  A  J,  0  and  A^^  x  A  f  oo. 

We  approximate  the  continuous  time  process  {W,z,y)  by  a  sequence  of  discrete  pa- 
rameter Markov  chains  {(H^,J,y^,  z^);n  =  1,2,  •••}  with  h  denoting  the  granularity  of 
the  grid  and  n  denoting  the  steps  of  the  Markov  chain.  For  every  h,  the  chain  has  the 
property  that,  at  each  step  n,  there  is  a  choice  between  investment  and  consumption.  At 
any  time,  a  chain  can  either  make  an  instantaneous  jump,  (AC  =  h),  or  follow  a  "random 
walk",  (AC  =  0),  to  the  neighboring  states  on  the  grid.  At  the  cutoff  boundaries  the 
chain  is  reflected  to  the  interior  in  a  manner  consistent  with  its  dynamics  in  the  interior 
of  the  domain.  Fix  a  chain  and  its  corresponding  grid  size  h.  The  transition  probabilities 
for  this  Markov  chain  are  specified  as  follows: 
1.  The  case  of  no  consumption — (AC  =  0): 

The  chain  can  possibly  move  from  the  current  state  {k,i,j)  only  to  one  of  the  four 
neighboring  states:  {k  +  l,i,j),  {k  —  l,t,j),  {k,i  —  l,j),  and  {k,i,j  —  1).  For  any 
investment  policy  A  =  I  y.  A,  the  transition  probabilities  in  this  case  are  defined  as 


P^[{Ki,Jlik,i-l,j)] 


P/1MVV2  +  kh^r  +  fiA) 

Q'{k,i,j) 
fc^/tMV/2  +  fc/t^(M) 
Q'{k,i,j) 


Q'{k,i,jy 


Pk[{k,iJ),{k,iJ)]    =     l-Pi^[ik,iJ),{k+hiJ)]-P;^[{k,iJ),ik-hhJ)] 

-P;^[{k,i,j),ik,i-lJ)]-Pl^[{k,ijy{k,iJ-l)] 
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where  the  normalization  fcLctor  (5''(^,t,i)  is  given  by 

Q''{k,iJ)  =  ^max  {Ph''{l  X  A)V2  +  kh^[r  +  I  x  A{(x  +  r)]  +  {\i  +.  ^j)h''}  . 

The  recipe  for  these  transition  probabiHties  is  a  slightly  modified  version  of  those  sug- 
gested by  Kushner  (1977).  Note  that  the  normalization  factor  is  independent  of  the 
investment  policy  A.  This  specification  implies  that,  at  each  step,  the  Markov-chain  has 
a  strictly  positive  probability  of  remaining  at  the  same  node. 

Furthermore,  we  define  the  incremental  difference  Ay^  =  j/^^^  —  y^,  Az^  =  z^^j  — 
z^J,  and  AWl^  =  W„\i  -  W^^.  The  quantity  At^{k,i,j)  is  equal  to  h?IQ^[k,i,i)  and 
interpreted,  following  Kusnher  and  Dupuis  (1992),  as  the  "interpolation  time"  for  the 
Markov  chain.  Note  that  the  interpolation  time  At^[k,i,j)  is  a  random  variable  that 
varies  from  state  to  state.  At  step  n  of  the  chain,  with  the  previously  defined  time  scale 
A<^,  one  can  verify  that 

^n'[Ay^]    =    -XyAt'^^-ALl 
£„^[Az„^]    =    -^zAt^^-ARl 

K\^K]     =     ^[r-  +  Mf^  -  r)Wu  -  A^n,  and  ^  °^ 

E^lAWl:  -  E^lAW!:]]''    =    H^MVAt;;  +  O(Aii), 

where  E^  denotes  expectation  conditional  on  the  nth-time  state  (M^,J,y^,  z^),  and  where 
the  reflecting  processes  AL^,  A/2^,  and  AZ)^  equal  to  the  positive  value  h  only  when 
y^,  z^,  and  Wj^  reach  their  respective  boundaries.  This  implies  that  the  first  and  second 
moments  of  the  Markov  chain  approximate  those  of  the  continuous  process  {Wt,y,,Zg). 
We  call  this  property  "local"  consistency  of  the  Markov  chain. 
2.  The  case  of  consumption —  (AC  =  h): 

The  chain  jumps  along  the  direction  (  —  1,  A,  ^)  from  the  current  state  {k,  i,j)  to  the  state 
{k  —  \,i  +  X,j  +  ^).  However,  the  later  state,  {k  —  l,i  +  \,j  +  0),  is  usually  not  on  the  grid 
except  in  the  trivial  case  A  =  1  =  ^.  For  ease  of  programming,  we  take  the  intersection  of 
the  direction  vector  {  —  \,\,^)  with  the  corresponding  surface  boundary  and  randomize 
between  three  grid  points  axijacent  to  the  intersection  point.  We  randomize  in  such  a 
way  that  the  expected  random  increment  will  be  along  the  direction  (  — l,A,/3)  and  of 
length  equal  to  the  distance  from  the  starting  state  to  the  point  of  intersection. 
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-  Without  loss  of  generality,  we  assume  hereafter  that  A  <  1  <  ^.  In  this  case,  the 
intersection  occurs  inside  a  triangle  spanned  by  the  three  grid  points  {k,i,j  +  1),  {k  — 
1,1,  j  +  1),  and  {k  —  1,1  +  l,j  +  1).  The  transition  probabilities  can  be  defined  as: 

0-1 


P^[{k,i,j),{k,iJ  +  l)]     = 


/?     ' 


Pk[{k,i,j),ik-'^,ij  +  l)]    =    -^,  and 
P^[{k,iJ),{k- 1,1 +  1J  +  1)]    =    ^, 
In  this  case,  also,  we  can  verify  the  property  of  "local"  consistency: 

E^fAy^:]  =  AAC,         E^[Az':i]  =  0AC,         and         E„^[AW^„^]  = -AC         (29) 

where  AC  =  4  is  the  increment  in  consumption.  The  quantities  in  the  right-hand 
side  of  the  equalities  in  (29)  are  the  respective  changes  in  the  continuous  time  process 
corresponding  to  AC  units  of  consumption. 

4.3     The  Markov— Chain  Decision  Problem 

A  policy  {A,C)  is  admissible  if  it  preserves  the  Mairkov  property  in  that 


conditional  on  {)Zvy,T,{~t<^''  I        I  A'^K*.',  j),  (*'.  i',  j')l,    if  AC  =  k  , 


where  W  =  kh  and  W  =  k'h,  y  =  ih  and  y'  =  i'h,  z  =  jh  and  z'  =  j'h.   The  control 
problem  for  the  discrete  parameter  Markov  chain  is  then  to  solve  the  program: 

J''(^,i,j)  =  rnax  ^t.  £  ^""""(yn,  ^n)Ai^  (30) 

71=0 

where  tjj  =  Xro</<n  ^'l*-  Note,  (30)  is  analogous  to  (26)  in  the  sense  that  the  sum  in  the 
former  approximates  the  expected  integral  in  the  latter. 
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The  discrete  dynamic  progreimming  equation  now  takes  the  iterative  form 

J'{k,i,j)     =    m^x{Zk'„',:'P^[{k,iJ)Ak\i\r)]J'{k',i',j'), 

maxo<KN.{e-*^''•('^•'■•^•)Ev..^;'^/:*[(t,t,;),(fc^^^;0]^'(fc^^^/^        (31) 
+U{iJ)At'{k,i,j)} 

for  {k,i,j)  G  Gn,  with  iterative  reflection®  at  the  artificial  boundaries,  where  we  recall 
that  Ph[*,  *]  and  P^[*,  *]  are  the  transition  probabilities  of  the  chain.  Now,  let  us  denote 

D-J{k^i,j)  =  nk,r,j)-nk,i-l,j)^       D-Jik^iJ)  =  Ak,t,j)-Hk,i,J-l)^ 

DfJiKrJ)  =  Ak,i+hj)-J{k,i,j)^       D-Jik,i,j)  =  Ak,iJ  +  l)-J{k,z,j)^ 

h  ^  h 

DjJ{k,i,j)  = ,       D^  J{k,t,j)  = ,  and 

n2   jf,    ■   ■.       Jjk  +  l,i,j)  -  2J{k,i,j)  +  J{k  -  l,ij) 
L>k  J\k,i,j)  =  -^ . 

Using  this  notation,  we  can  express  the  discrete  Bellman's  equation  (31)  in  the  form: 

~  L     At^{k,i,j) 
XDtJ''{k-l,i,j  +  l)  +  l3Dp\k,iJ)-D;j''{k,iJ  +  l)     <    0,  where       (33) 

C^  =  Ih/MVD^  +  W{r  +  Ati)Dt  -  WArD;  -  XyD-  -  ^zDj. 

Furthermore,  one  of  these  two  inequcilities  must  hold  as  an  equality  at  each  {k,  i,j)  €  Gh- 
In  particular,  if  it  is  optimal  not  to  consume  at  {k,i,j)  then  (32)  holds  as  an  equality. 
Otherwise,  (33)  holds  as  an  equality  at  {k,ij).  Clearly,  [1  -  e-*^'''(*''-')]/Af''(Jt,z', j) 
approximates  the  discount  factor  6  as  h  I  0.  This  leads  to  a  pair  of  discrete  differential 
inequalities  which  gives  a  finite-difference  approximation  of  Bellman's  equation  (31)  for 
the  constrained  maximization  problem  on  the  domain  [0,  Mw]  x  [0,My]  x  [0,Mi].  In 
section  5,  we  outline  a  proof  that  the  above  described  sequence  of  finite-difference  ap- 
proximations, together  with  the  boundary  specifications,  converges  to  the  solution  of  the 
continuous  time  Bellman's  equation. 


J*"  +  m&x{Ctrik,iJ)]  +  U{i,j)    <    0,  (32) 


^This  specification  will  be  detailed  in  the  end  of  this  section. 
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4.4      Boundary  Specification  of  The  Value  Function 

■-  For  numerical  implementation,  we  attach  a  "fictitious"  surface  to  each  cutoff  boundary 
surrounding  the  cubic  grid  Gh  in  which  the  Markov  chain  is  defined.  We  denote  by 
G^  the  grid  including  this  attached  exterior  surfaces.  The  three  surfaces  {W  =  0,z  = 
0,  andy  =  0)  are  natural  boundaries  whereas  the  three  surfaces  {W  =  Mw  +  h,z  = 
Mz  +  h,  and  y  =  My  +  h)  are  artificial  fictitious  boundaries.  The  value  function  on  the 
artificial  boundaries  will  be  specified  to  capture  the  optimal  policy  in  a  manner  locally 
consistent  with  that  of  the  continuous  processes  at  the  boundary  surfaces.  These  are  the 
conditions  we  turn  to  next. 

•  At  the  boundary  W  =  0,  the  boundajy  condition  is  given  by  the  natural  constraint 
J'{0,ij)  =  ^iih,jh). 

•  At  the  boundary  surfax:e  y  =  0,  it  is  clecir  that  the  optimal  policy  calls  for  con- 
sumption because  of  infinite  marginal  utility  for  y  on  that  boundary.  Hence,  the 
value  function  at  this  boundary  is  given  by: 

J'{k,0,j)  =  ^J'{k,0,j  +  1)  +  ^A^  -  1,0,;  +  1)  +  jJ'ik  -  1,1,;  +  1). 

•  Similarly,  at  the  boundary  2  =  0,  because  of  infinite  marginal  utility  for  z,  imme- 
diate consumption  is  optimal.  Hence,  the  value  function  satisfies: 

J'{k,i,0)  =  i^jf^{k,i,l)  +  ^J'{k  -  l,i,  1)  +  ^J'{k  -l,i  +  l,  1). 

Next  consider  the  artificial  cutoff  boundaries. 

•  At  the  top-level  boundary  W  =  M\y  +  h,  we  take  it  that  the  wealth  level  is  so  high 
that  the  agent  will  optimally  choose  to  consume  regardless  of  the  levels  of  z  or  y. 
As  a  result,  the  iterative  scheme  of  the  value  function  at  that  boundary  is  given  by 
the  natural  constraint  XDf  J''  +  I^Dfj''  ~  DtJ*"  =  0. 
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•  At  the  y-side  boundaxy  y  =  My  +  h,  we  let 

DfJ^  =  0,        if  AC  =  0  and 
/3D/ J''  -  DtJ''  =  0,       if  AC  =  h. 

•  At  the  z-side  boundary  z  =  M^  +  /i,  we  let 

Dp''  =  0,        if  AC  =  0  and 
XDfJ''  -  DtJ^  =  0,       if  AC  =  h. 

5      Convergence  of  the  Numerical  Scheme 

In  this  section,  we  analyze  the  convergence  of  the  value  function  and  the  optimal  con- 
trols in  the  approximating  Markov  chain  problems  to  those  in  the  original  continuous 
time  problem.  We  first  analyze  the  properties  of  the  finite-difference  approximation 
scheme  described  in  section  4.  In  particular,  we  show  that  the  discrete  Markov  chain 
approximation  approach  produces  a  monotone,  stable,  and  consistent  finite-difference 
approximation  to  the  Bellman's  equation  that  characterizes  the  value  function. 

Armed  with  these  results,  we  proceed  to  prove  that,  as  the  mesh  size  goes  to  zero 
and  the  size  of  the  cubic  domain  Qm  increases,  the  sequence  of  value  functions  in  the 
discrete  Markov  chain  problems  converges  to  the  value  function  of  the  original  problem 
uniformly  on  any  compact  subset  of  the  domain  Q.  Furthermore,  convergence  obtains  re- 
gardless of  the  boundary  conditions  specified  on  the  artificial  boundaries  of  the  restricted 
domains  Qm-  The  proof  is  based  on  the  fact,  we  already  established,  that  the  value 
function  J  is  the  unique  viscosity  solution  of  Bellman's  equation  (27).  In  the  classical 
case  with  a  smooth  value  function  (see,  for  instance,  Strikwerda  (1989)),  convergence 
follows  directly  from  consistency  and  stability  of  the  finite-difference  scheme.  Here,  the 
situation  is  different  because  we  do  not,  in  general,  have  a  classical  solution  of  Bellman's 
equation.  We,  hence,  utilize  the  comparison  principle  for  subsolutions  and  supersolutions 
of  Bellman's  nonlinear  partial  differential  equation  in  order  to  gu£u-antee  convergence  of 
the  approximations  to  the  value  function  of  the  continuous-time  problem. 
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5.1      Properties  of  the  Finite— Difference  Scheme 

Let  B{Gh)  denote  the  spa<:e  of  real-valued  functions  on  the  extended  lattice  Gh-  To  study 
the  discrete  Bellman's  equation  (31),  we  introduce  the  mappings 

T^:    B{Gh)-^B{Gh)         and         Sh  :    B{Gk)  ^  B{Gk)     where 
{T;^V)ik,i,j)    =    e-'^'"^"''^^  Y:  P,[{k,iJ),{k\i'J')]Vik',t'J'),         VA 

k',i',j' 

{S,V){k,iJ)    =      Yl  PK[{k,i,j)Ak',i\j')]V{k',i',j'), 

k',i',j' 

for  any  V  G  B{Gh)-    In  the  case  of  no  consumption,  we  set  Ak  =  m\nk,i,j  At^{k,i,j). 
Obviously  A/,  >  0.   Define  the  operator  norm  ||  •  ||  as  HrH  =  sup^gg  ||Tu||oo/||u||oo5  with 
II  ■  lloo  being  the  L°°-norm  of  the  bounded  function  space  B. 
The  discrete  Bellman's  equation  can  be  rewritten  as  follows: 

J\k,ij)  =  max{max[T;f  J''(fc,t-,j)]  +  u{iJ)At\k,iJ),SHJHk,iJ)},     V(^,t, ;)  €  Gh. 

(34) 
It  is  clear  that  the  solution  of  the  discrete  Bellman's  equation  is  a  fixed  point  of  the 
operator  equation  (34).  The  properties  of  the  operators  T/^  and  5a,  which  are  useful  for 
proving  convergence,  are  recorded  in  the  following  proposition. 

Proposition  2  7^* '5  are  strict  contraction  operators  with  \\Tf^\\  <  e"*^*"  <  1  and  Sh  is 
a  contraction  operator  with  \\Sh\\  <  1.  Thus,  for  every  h,  there  exists  a  function  J^  that 
solves  (34)- 

Proof.   We  estimate  directly 

\T,^V{k,i,j)-T;^V'ik,i,j)\    <    e-^^'''('=-'"-^)  Y:  PH[ik,tJ)Ak\i'J')]\V{k\i\j')-V'{k',t\j')\ 

k'.i'j' 

<  e-*^"     max     \V{k,iJ)-V'{k,i,j)\     and 

(*,i,»eGh 

\ShV{k,iJ)-SHV'ik,iJ)\    <      Y   Ph[{k,iJ),{k',i\j')]\V{k',i'J')-V{k',i'J')\ 

k',i',j' 

<  max     \V{k,iJ)-V'ikJ,j)\ 
(*,t,j)eGh 
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which  implies  the  property  of  contraction.  Given  that  both  Tf^  and  Sh  are  both  contrac- 
tions, the  second  assertion  follows.       I 

Next,  we  discuss  the  properties  of  the  finite-difference  scheme,  (32)  and  (33),  associ- 
ated with  the  Markov  chain  approximation.  On  a  grid  of  size  h,  we  will  denote  a  given 
finite-difference  operator  by  Ch  and  interpret  ChV  to  be  the  result,  at  each  grid  point, 
of  applying  the  scheme  to  the  function  v.  We  will  that  v  <  v'  \i  v{W,y,z)  <  v'{W,y,z) 
at  all  {W,y,z)  €  Q.  Recall  that  a  finite  difference  scheme  is  said  to  be  monotone  if 
^hV  ^  jC/iu'  when  v  <  v'.  The  scheme  ChV  =  /  is  said  to  be  consistent  with  the  partial 
differential  equation  Cv  =  f  if,  for  ciny  smooth  function  <f>,  we  have  £/,</>  —  C4>  — >  0, 
pointwise  at  each  grid  point,  as  /i  J.  0.  Finally,  a  finite-difference  scheme  Ch.v^  =  /  is 
said  to  be  stable  if  there  exists  /iq  >  0  such  that  ||t;''l|t»  ^  K,  for  all  0  <  /i  <  /iq,  for  any 
compact  domain  T)  and  some  positive  constant  K  independent  of  h,  where  |1  •  ||p  denotes 
the  L°°('D)-norm.  The  following  result  establishes  the  properties  of  the  finite-difference 
scheme  we  constructed  in  section  4.2. 

Theorem  3    The  discTtte  Markov  chain  approximation  yields  a  monotone,  stable,  and 
consistent  finite-difference  scheme  for  Bellman's  equation. 

Proof.  First,  monotonicity  of  the  scheme  is  implied  by  monotonicity  of  the  discrete 
operators  T/^V  <  T;^V'  and  ShV  <  ShV  for  any  V,V'  G  B{Gh)  such  that  V  <  V. 
Second,  we  Cein  verify  directly  that,  for  any  smooth  function  <^,  and  with  the  obvious 
notation: 

Tl^4>\k,i,j)-<f>\Ki,j)    =  -[l-e-'''''^''''^^)]4>'{k,i,j)  +  CU'{k.ijW{k,i,j), 
Sh<t>'{k,i,j)  -  <l>'{k, i,j)     =  [XDr<f>'{k  -  l,i,  j  +  1)  +  ^Dl<f>\k,i,j)  -  D;4>\k,i,j  +  l)]h, 

(35) 

where  <i>'*{k,i,j)  =  ^{kh,ih,jh).  The  difference  terms  can  be  estimated  by 

Df(i>  =  4>w±  ^<f>ww  xh  +  C>(/i') ,       Dl<f>  =  4>ww  +  0(/i') , 
D;  (i>  =  <i>y -^ -4>yy  X  h  +  0{h^) ,  a.nd        Dj  4>  =  (f),  + -<f>,,  xh  +  0{h^). 
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Therefore,  £'^<f>  —  €■*<!>  =  0{h),  which  implies  the  consistency  of  our  finite-difference 
approximation.  Finally,  it  follows  from  Proposition  2  that,  for  any  investment  policy  A, 
the  set  {(T;^,  5/,);  V/i}  is  uniformly  bounded  in  operator  norm,  which  implies  the  stability 
of  the  finite-difference  scheme.      I 

Note  that  consistency  as  used  in  theorem  3  is  actually  equivalent  to  the  "local" 
consistency  property  (28)  and  (29)  of  the  corresponding  Markov  chain. 

5.2      Convergence  of  The  Value  Function 

In  this  section,  we  report  the  result  that  the  sequence  of  value  functions  in  the  approxi- 
mating decision  problems  converges  to  the  Vcdue  function  of  the  original  continuous  time 
program.  The  proof  of  this  result  requires  special  attention  to  the  gradient  constraints 
in  the  sequence  of  Bellman's  equations  of  the  approximating  control  problems. 

Theorem  4  (Convergence)  The  sequence  of  solutions  J'^ik,  i,j)  of  the  discrete  Markov 
chain  problems  converges  to  the  value  function  J{W,y,z)  of  the  continuous-time  problem 
uniformly  on  any  compact  subset  of  Q  =  (0,  oo)"'  as  h  [  0  and  Nw  T  oo?  ^z  T  °°>  '^^^ 
Ny  t  oo,  in  such  a  way  that  hk  —*  W,  hi  —*  y,  and  hj  —*  z. 

Proof.   For  each  (W^,y,  2)  6  Q,  define 

r{W,y,z)    =         limsup       J^{W\y\z'),     and  (36) 

(W,v',x')-(V*',»,x) 
fciO,  (W',v',i')gG'> 

J.(iy,y,z)    =     ,^,liminf,       j\W\y\z').  (37) 

'>10.(W",>i',j')€G'' 

J'  is  finite  at  every  point  (VF,  j/,2)  since  the  finite-difference  scheme  is  stable.  Clearly, 
J'  is  upper  semicontinuous  ajid  J,  is  lower  semicontinuous,  and 

J>{W,y,z)<J'{W,y,z),   on  Q.  (38) 

Note  that  J''(0,  y,  2)  =  'I'(y,  2),  V(y,  2)  G  G^.  Therefore,  J'(0,  y,  2)  =  ^{y,  2)  =  J.(0,  y,  z) 
for  {y^z)  G  (0,00)  X  (0,oo).  We  will  verify  here  that  J*  is  a  viscosity  subsolution  and  J» 
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is  a  viscosity  supersolution  of  Bellman's  equation 

max{-<5J+max{£^J}+u(j/,2),AJy+^Jj-JH'}  =0,     on  Q     with  J(0,y,z)  =  *(y,-2). 

(39) 
It  then  follows  from  the  comparison  result  in  theorem  2  that  J*  <  J,  on  Q.  This  result, 
together  with  (38),  implies  that  J'  =  J.,  and  hence,  by  uniqueness,  both  are  equal  to  J. 
Let  {Wo,  yo,  zo)  G  Q  be  a  strict  local  mjiximumof  J'  ~(f>  for  some  (j)  €  C^'^'^{Q).  Since 
we  take  the  limit  /i  I  0  and  {Nw-,  Ny,  N^)  t  cxd  in  such  a  way  that  h  x  {Nw,  Ny,  N^)  ]  oo, 
there  exists  some  ho  >  0  such  that  for  h  <  ho,  {Wo.,yo,2o)  is  an  interior  point  of  the 
domain  spanned  by  the  grid  Gh-  By  the  property  of  "limsup",  there  exists  a  sequence 
indexed  by  h  such  that 

{Wh,yh',Zh)  -^  {Wo,yo,zo),         J''{W'h,yh,Zh)  -^  J'{Wo,yo,ZQ)         ash  10. 

For  h  <  ho,  J^  —  4>  hais  aloccJ  maximum  at  {Wh.,yh-i  Zh),  and  hence,  for  some  neighborhood 
A/'o  of  (VVo,2o,yo), 

J\W,,  y,,  z,)  -  4>{Wf,,  y,,  zh)  >  J\W,  y,  z)  -  4>{W,  y,  z),      if  {W,  y,  z)  6  A^o  .        (40) 

Since  J''  is  a  solution  of  the  discrete  operator  equation 

j''iWH,yk,ZK)  =  m^x{m^x[T^j''{WH,yH,ZH)]  +  u{yH,ZH)At\S'HJ\WH,yH,Zh)}, 

using  (40)  and  the  monotonicity  of  Tf^  and  Sh,  we  can  replace  J^  by  <!>  such  that 

max{max[T^^(^  -  <f>]{Wh,  y/.,  ^0  +  ^Vh,  ZK)At\  [5/.<A  -  <^]{Wh,  yn,  ^O}  >  0.         (41) 

On  the  other  hand,  we  recall  from  (35)  that 

T;^<i>-<f>    =    -[l-e-^^''']<P  +  C^4>At\     and  (42) 

SH<f>-<f>    =    [XDt4>  +  0Dl<f>-D:<f>]h.  (43) 

Therefore,  dividing  (42)  by  Af''  and  (43)  by  h,  the  inequality  (41)  at  {Wh,yh,  2/1)  becomes 
max{-( ^^r— )^  +  max[£;f  .?i]  +  u(y,,  z,),  AD.+  ^  +  /3Dlcf>  -  D;4>}  >  0, 
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for  h  <  ho-  Finally,  by  the  consistency  of  our  fmite-diiference  scheme,  as  we  let  /i  j  0, 
we  obtain 

max\^-S<f>  +  max[£'^(^]  +  u(yo,  zq),  X<f>y  +  ^(f>^  -  4>w]  >  0,    at  (Wq,  j/o,  -^o)- 

Thus,  J'  is  a  viscosity  subsolution  by  Definition  1. 

Using  similar  reasoning,  we  can  show  that  J,  is  a  viscosity  supersolution.  By  the 
comparison  principle,  theorem  2,  it  follows  that  J'{W,y,z)  <  J,(\V,y,z)  on  Q  which, 
combined  with  (38),  proves  that,  for  all  {W,y,z)  6  Q, 

Hm  j\W',y',z')  exists  =  J{W,y,z)  =  J'{W,y,z)  =  J,{W,y,z).     I 

hlO,  (VV",v',i')6G'' 

In  the  argument  of  convergence,  we  do  not  require  knowledge  of  the  particular  spec- 
ifications on  the  cutoff  boundaries  for  the  Mao-kov  chains.  The  same  limit  for  J^  will 
be  obtained  regardless  of  the  conditions  chosen  at  the  cutoff  boundaries.  This  results 
from  the  fact  that,  in  the  theoretical  sequence  of  approximations,  as  the  mesh  size  de- 
creases, the  size  of  the  corresponding  bounded  domain  increases.  However,  in  the  actual 
numerical  implementation,  the  size  of  the  restricted  domain  is  kept  fixed  and  hence  the 
choice  of  the  boundary  conditions  affects  the  quality  of  the  numerical  solution.  The 
boundary  specifications  discussed  in  the  previous  sections  were  chosen  to  obtain  a  good 
approximation  to  the  solution  of  the  continuous-time  problem. 

5.3      Convergence  of  Approximating  Policies 

Existence  of  the  value  function  and  the  feasibility  of  computing  its  values  with  high 
accuracy  do  not  guarantee  existence  of  optimal  policies  that  achieve  the  value  function. 
Contrast  this  with  classical  feedback  control  case.  In  that  case,  optimal  controls  can  be 
expressed  as  explicit  functions  of  the  value  function  and  its  derivatives.  As  a  consequence, 
existence  of  a  smooth  value  function  immediately  implies  existence  of  optimal  controls. 
In  our  case,  with  a  free-boundary  problem,  there  is  no  direct  relationship  between  the 
value  function  and  the  optimal  controls,  which,  if  exist,  are  not  feedback.  Furthermore, 
the  value  function  is  not  known  to  be  smooth. 
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In  this  section,  we  prove  convergence  of  the  optimal  investment  policies  in  the  approx- 
imating Markov  chains  to  the  optimal  investment  policy  of  the  diffusion  control  problem 
if  the  latter  indeed  exists.  We  will  parameterize  the  sequence  of  approximating  problems 
by  h  and  denote  the  optimal  consumption  and  investment  policies  in  problem  h  by  C'^ 
and  A''*,  respectively.  We  define  an  interpolation  of  the  value  function  for  the  Markov 
chain  problem  using  the  following  linear  aifine  interpolation: 

j\W,y,z)    =    j'{k,iJ)  +  DtJHk,i  +  lJ  +  l)x{W-kh) 

+D-J\k,ij  +  1)  X  (y  -  ih)  +  D-J\k,i,j)  X  {z-jh) 

for  kh  <  W  <  {k  -h  \)h,  ih  <  y  <  {i  +  l)h,  and  jh  <  W  <  {j  +  \)h;  with  appropriate 
extensions  at  the  cutoff  boundaries.  Clearly,  J'*{W,  y,  z)  is  continuous  on  the  cubic  region 
[0,  Mw]  X  [0,  Mj,]  X  [0,Mj].  One  can  easily  show  the  following 

Lemma  1  As  a  function  of{W,y,z),  J^{W,y,z)  is  nondecreasing  in  {W,y,z)  and  con- 
cave in  W  on  Qm  =  [0,  M]^. 

Proof.  The  proof  that  J'^  is  nondecreasing  in  its  arguments  follows  the  same  reasoning 
as  the  proof  of  property  1  of  the  value  function  in  Fitzpatrick  and  Fleming  (1991,  page 
827).  Furthermore,  the  proof  of  concavity  of  J''  in  W  follows  the  proof  of  property  2 
(page  829)  in  the  same  reference.      I 

Next,  we  utilize  the  special  monotonicity  and  concavity  features  of  the  problem  to 
prove  convergence  of  the  first  and  second  order  derivatives  dJw{k,  i,j)  — >  Jw{W^,  y,  ^)  and 
dJwwi^-ihJ)  ~^  Jww{^,y,z)  as  /i  J.  0  and  {hk,hi,hj)  — >  {W,y,z).  This  information 
is  used  to  show  that  the  optimal  investment  A'^  for  the  discrete  Markov  chain  control 
problem  converges  to  the  optimal  investment  policy  A'  for  the  continuous-time  model 
when  the  latter  exists. 

Theorem  5  Suppose  that  there  exists  an  optimal  control  policy  {A',C')  for  the  original 
continuos  time  problem..  Then,  the  sequence  of  policies  A'^{k,i.,j)  converges  at  all  points 
{W.,y.,z)  where  Jw  and  Jww  exist,  which  is  almost  everywhere,  to  the  optimal  investment 
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policy  A'{W,y,z)  as  h  I  0,  and  Nw  1  oo,Nz  1  oo,  and  Ny  ]  oo,  in  such  a  way  that 
hk  — >  W,  hi  —y  y,  and  hj  —*  z. 

Proof.  Using  standard  arguments,  we  can  show  that  the  value  function  J  is  increasing 
and  concave  in  its  arguments.  It  then  follows;  see  Royden  (1988,  §5.5),  that  J  has  first 
and  second  derivatives  almost  everywhere.  As  a  result,  since  J  is  a  viscosity  solution  of 
Bellman's  equation,  J  satisfies  that  equation  in  almost  everywhere  sense.  Carrying  out 
the  maximization  in  (7),  we  conclude  that  A'  =  j—-^'^  almost  everywhere. 

We  rewrite  the  "finite-difference"  form  of  the  Bellman's  equation  for  the  interpolated 
value  function  J^ 

1  _     -SAt" 
[—^-^]J'{k,tJ)  +  0{h)  =  ^ma^^{£if /"(/:,  i,;)}  +  f/(i,j),  (44) 

in  the  case  of  no  consumption  AC  =  0.  From  this,  we  can  compute  the  optimal  invest- 

ment  policy  by  the  formula  A'  {k,  i,j)  = wa^D^J^>^ — '  ^°^  (^'  ^'■?)  ^  ^  ~  *^^  interior 

grid.  Note  that  only  the  first  and  second  order  derivatives  of  J,  with  respect  to  W,  are 
used  in  that  formula.  Substituting  the  previous  formula  back  into  the  difference  equation 
(44),  and  after  a  long  calculation,  we  can  express  DlJ'^  as  H{Dfj'^,  D~  J^,  Dj  J^i  J^)  for 
some  continuous  function  H.  Now,  using  arguments  similar  to  those  in  Fitzpatrick  and 
Fleming  (1991,  theorem  4),  and  lemma  1  ,  we  can  prove  that  DfJ'^  —^  Jw,  D~  J^  — »  Jy, 
and  D~ J^  —^  J^.  As  a  result,  D\J^  — +  Jww  and  consequently  we  must  have  that 
A'^  — >  A',  at  the  points  of  differentiability  with  respect  to  W ,  as  /i  |  0  and  as  the  size 
of  the  domain  Qm  increases  to  Q.       I 

Finally,  we  provide  the  following  result  that  shows  that  the  optimal  "regions"  of  con- 
sumption and  abstinence  in  the  approximating  sequence  converge  to  the  corresponding 
regions  in  the  original  problem.  Specifically,  if  at  a  given  point  (iy,y,z),  the  optimal 
solution  of  the  original  problem  prescribes  a  jump  in  consumption,  then,  all  the  approxi- 
mating optimal  solutions,  except  for,  at  most,  a  finite  number,  also  prescribe  jumps  at  the 
grid  points  that  converge  to  {W ,  y,  z).  Similarly,  if  the  optimal  solution  in  the  continuous 
time  problem  calls  for  no  consumption  at  {W,y,z),  then  almost  all  the  approximating 
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optimal  solutions  will  also  caJl  for  no  consumption  at  the  grid  points  that  converge  to 

Theorem  6  Suppose  that  there  exists  an  optimal  control  policy  (A'^C)  for  the  original 
continues  time  problem.  Consider  the  sequence  of  approximating  problems  indexed  by  h 
and  let  h  I  0,  and  Nw  T  °°i  ^z  T  °°>  <*"<^  ^y  T  oo-  Let  x  =  {W,  y,z)  €  Q  be  such  that  the 
value  function  J  is  twice  continuously  differentiate  in  a  neighborhood  of  x.  Let  AC(x) 
be  the  size  of  the  optimal  consumption  jump  at  x.  Let  {x^}  =  {(fc, i,j)}  be  a  sequence 
of  grid  points  that  converges  to  x  in  such  a  way  that  J'^{x^)  —^  J{i)-  If  AC*{x)  >  0 
(AC'{x)  =  O;,  then  AC*''(x'')  >  0  (AC'\x^)  =  0),  respectively,  for  all  h,  except  for,  at 
most,  a  finite  number. 

Proof.  Consider  the  case  when  AC*(i)  =  0.  Since  J  is  smooth  in  a  neighborhood  of  x,  it 
follows  from  arguments  similar  to  those  used  for  proving  the  verification  result — theorem 
2—  in  HH&Z  that 

pj,  -\-\Jy-  Jw  <Q   at  X.  (45) 

Now  suppose  that  AC*''(x'')  =  0  for  only  a  finite  number  of  points  in  the  sequence. 
It  then  follows  from  Bellman's  equation  for  the  Markov  chain  decision  problem  that 
/3D/J''  +  XDfJ''  -  DtJ''  =  0  a.t  x^,  for  all  but  a  finite  number  of  elements  in  the 
sequence.  However,  from  the  proof  of  theorem  5,  we  conclude  that 

lim(/3£>t  J''  +  XDtJ''  -  DtJ''){x^)  =  {^Jz  +  XJy  -  ^)(x)  =  0 

which  contradicts  (45).  The  proof  of  the  case  when  AC*(x)  >  0  follows  the  same  argu- 
ments and  utilizes  the  result  of  theorem  5  that  {A'''}  converge  to  A".       I 

6      Concluding  Remarks 

The  analysis  of  this  paper  achieves  its  objective  of  providing  technical  support  for  the 
numerical  solution  reported  in  HH&Z.  In  particular,  we  provide  proofs  of  convergence 
of  the  sequences  of  both  the  vaJue  functions,  the  optimal  investment  policies,  and  the 
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optimal  consumption  regions  in  the  approximating  problems  to  the  corresponding  entities 
in  the  original  problem.  Such  results  justify  the  description  in  HH&Z  of  the  numerical 
results  as  e-optimal  solutions.  There  are  still  some  interesting  theoretical  open  questions. 
We  have  not  proved  existence  of  an  optimal  solution  in  the  class  of  admissible  controls. 
Furthermore,  we  are  also  quite  silent  on  the  reguljirity  properties  of  the  free-boundary. 
We  hope  that  further  research  in  this  area  sheds  some  light  on  these  issues. 

We  remark  that  there  is  another  method  of  proving  convergence  of  the  sequence  of 
value  functions  in  the  approximating  Markov  decision  problems.  That  method  is  based 
on  probabilistic  techniques  from  the  theory  of  weak  convergence  of  probability  measures. 
For  a  detailed  exposition  of  the  probabilistic  approach,  we  refer  the  reader  to  Kushner 
and  Dupuis  (1992,  §9  — §10)  and  the  references  therein.  The  idea  there  is,  very  roughly,  as 
follows.  One  constructs  continuous  time  interpolations  of  the  optimally  controlled  Markov 
chains.  Then  one  proves  that  the  sequence  of  interpolated  processes  has  a  subsequence 
that  converges  weakly  to  an  optimally  controlled  process  of  the  original  problem.  Since 
the  approximating  sequence  has  the  same  objective  functional  as  the  original  problem,  one 
then  concludes,  using  the  theory  of  weak  convergence,  that  the  sequence  of  approximating 
value  functions  converges  to  the  value  function  of  the  original  problem. 

Furthermore,  by  extending  the  space  of  admissible  controls  to  the  so-called  "relaxed 
controls";  see  Kushner  and  Dupuis  (1992,  §4.6),  one  can  show  that  there  exists  an  optimal 
relaxed  control  for  the  original  problem.  The  optimal  relaxed  controls,  however,  need  not 
be  in  the  class  of  original  admissible  controls.  In  any  case,  the  sequence  of  approximating 
optimal  controls  converges  weakly  to  the  optimal  relaxed  control  of  the  original  problem. 
We  chose  not  to  present  the  probabilistic  approach  here  because  the  notion  of  weak 
convergence  does  not  guarantee  that  the  optimeil  relaxed  control  are  adapted  to  the 
information  structure  in  the  economy.  As  a  result,  the  optimal  relaxed  control  may  fail 
to  be  feasible  and  admissible  control  policies. 

We  remark,  however,  that  using  the  analytical  approach  we  can  prove  that  the  ap- 
proximating optimal  investment  policies  and  the  consumption  regions  converge  pointwise 
to  their  corresponding  counterparts  in  the  original  problem  if  the  latter  exists  in  the  class 
of  admissible  controls.  This  is  a  stronger  result  than  weak  convergence  to  a  relaxed  con- 
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trol,  which  may  not  be  in  the  class  of  the  original  admissible  controls,  that  obtains  using 
probabilistic  techniques.-  Finally,  we  note  that  neither  approach  establishes  existence  of 
optimaJ  policies  in  the  originad  admissible  class  of  controls  described  in  section  2.2.  Es- 
tablishing such  a  result  requires  information  about  the  regularity  of  the  free-boundary. 
For  more  on  this  issue,  we  refer  the  reader  to  Soner  and  Shreve  (1989)  and  Hindy  and 
Huang  (1993). 

7     References 

1.  M.  Crandall  and  P.  L.  Lions,  Viscosity  Solutions  of  Hamilton- Jacobi  Equations, 
Transactions  of  the  AmericeLD  Mathematical  Society  277,  1982,  pp.  1-42. 

2.  M.  G.  Crandall,  H.  Ishii,  and  P.  L.  Lions,  User's  Guide  to  Viscosity  Solutions  of  Sec- 
ond Order  PartiaJ  Differential  Equations,  Bulletin  of  the  American  Mathematical 
Society,  1992. 

3.  B.  Fitzpatrick  and  W.  Fleming,  Numerical  Methods  for  Optimal  Investment-Consumption, 
Mathematics  of  Operations  Research  16,  1991,  pp.  823-841. 

4.  W.  Fleming  and  H.  Soner,  Controlled  Markov  Processes  and  Viscosity  Solutions, 
Springer- Verlag,  New  York,  1993. 

5.  W.  Fleming  and  P.  Souganidis,  On  The  Existence  of  Value  Functions  of  Two- 
Player,  Zero-Sum  Stochastic  Differential  Games,  Indiana  University  Mathematics 
Journal  38,  1989,  pp.  293-314. 

6.  A.  Hindy,  C.  Huang,  and  H.  Zhu,  Optimal  Consumption  and  Portfolio  Rules  With 
Durability  and  Habit  Formation,  Working  Paper,  Stanford  University,  1993. 

7.  A.  Friedman,  Partial  Differential  Equations  of  Parabolic  Type,  Prentice-Hall,  Inc., 
New  York,  1964. 


7    REFERENCES  35 

8.  A.  Hindy  and  C.  Huang,  Optimal  Consumption  and  Portfolio  Rules  With  Durability 
and  Local  Substitution,  Econometrica  61,  1993,  pp.  85-122. 

9.  R.  Holmes,  Geometric  Functional  Analysis  and  Its  Applications,  Springer-Verlag, 
New  York,  1975. 

10.  H.  J.  Kushner,  Probability  Methods  for  Approximations  in  Stochastic  Control  and 
for  Elliptic  Equations,  Academic  Press,  New  York,  1977. 

11.  H.  J.  Kushner  and  P.  G.  Dupuis,  Numerical  Methods  for  Stochastic  Control  Prob- 
lems in  Continuous  Time,  Springer-Verlag,  New  York,  1992. 

12.  O.  A.  Ladyzenskaja,  V.  A.  Solonnikov,  and  N.  N.  Ural'ceva,  Linear  and  Quasilinear 
Equations  of  Parabolic  Type,  Academic  Press,  New  York,  1968. 

13.  P.  L.  Lions  and  P.  E.  Souganidis,  Differential  Games,  Optimal  Control,  and  Direc- 
tional Derivatives  of  Viscosity  Solutions  of  Bellman's  and  Isaac's  Equations,  SIAM 
Journal  on  Control  23,  1985,  pp.  566-583. 

14.  P.  L.  Lions,  Optimal  Control  of  Diffusion  Processes  and  Hamilton-Jacobi-Bellman 
Equations,  Part  II:  Viscosity  Solutions  and  Uniqueness,  CommuDications  in  Pure 
and  Applied  Matbematic's  37,  1983,  pp.  511-553. 

15.  H.  L.  Royden,  Real  Analysis,  Third  Edition,  Macmillan  Publishing  Co.,  New  York, 

1988. 

16.  H.  M.  Soner  and  S.  E.  Shreve,  Regularity  of  the  Value  Function  of  a  Two-Dimensional 
Singular  Stochastic  Control  Problem,  SIAM  Journal  on  Control  and  Optimization 
27,  1989,  pp.  876-907. 

17.  J.  C.  Strikwerda,  Finite  Difference  Schemes  and  Partial  Differential  Equations, 
Wadsworth  &  Brooks,  California,  1989. 


7    REFERENCES  36 

18.  T.  Zariphopoulou,  Investment-Consumption  Models  with  Transaction  Fees  and 
-        Markov  Chain  Paramieters,  SIAM  Journai  on  Control  and  Optimization  30,  1992, 

pp.  613-636. 

19.  H.  Zhu,  Variational  Inequalities  and  Dynamic  Programming  for  Singular  Stochastic 
Control,  Doctoral  Dissertation,  Division  of  Applied  Mathematics,  Brown  Univer- 
sity, 1991. 


2935    j6i. 


MIT   LIBRARIES 


3  lOaO  D0flSbSE4  1 


Date  Due 


